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ABSTRACT 


The  Australian  Theatre  (AST)  exercises  operational  command  for  the  planning  and  conduct  of 
Defence  operations,  activities  and  campaigns.  In  2001  the  Chief  of  the  Defence  Force  (CDF) 
and  the  Defence  Science  and  Technology  Organisation  (DSTO)  held  a  series  of  roundtable 
discussions.  Amongst  other  things  they  identified  Theatre  evaluation  as  a  key  issue  for 
supporting  the  development  of  AST.  The  recurring  theme  was  the  need  to  better  understand 
current  Theatre  performance  and  to  use  the  principles  of  ‘Learning  and  Experimentation’1  as  the 
means  for  development.  As  a  result,  a  team  from  DSTO2  and  Headquarters  Australian  Theatre 
(HQAST)3  conducted  a  study  into  the  feasibility  of  developing  a  methodology  for  evaluation  and 
conducting  a  baseline  measurement  of  the  Theatre.  This  paper  reports  on  the  feasibility  and 
initial  development  of  the  Enterprise  Evaluation  Methodology  (EEM)  which  takes  the  approach 
of  decomposing  Australian  war  fighting  principles  into  performance  goals  for  the  Operational 
Headquarters.  Despite  the  resource  intensive  nature  of  the  particular  approach  taken,  the  Theatre 
Baseline  approach  was  judged  ahead  of  current  practice  because  it  was  based  on  values  and 
principles  rather  than  process. 

1  Introduction 

AST  exercises  operational  command  for  the  planning  and  conduct  of  Defence  operations, 
activities  and  campaigns.  Discussions  between  the  Chief  of  the  Defence  Force  and  the  Defence 
Science  and  Technology  Organisation  identified  evaluation  of  AST  as  a  key  component  to 
support  the  development  of  the  Theatre.  As  a  result,  a  team  from  DSTO  and  AST  conducted  a 
study  into  the  feasibility  of  developing  a  methodology  for  evaluation  and  conducting  a  baseline 
measurement  of  the  Theatre. 

Developing  and  measuring  a  baseline  is  a  fundamental  activity  to  support  planned  development 
and  change  in  the  Theatre.  A  baseline  establishes  the  current  performance  of  the  Theatre  using 
both  quantitative  and  qualitative  measures.  A  baseline  may  also  provide  a  comparative  measure 
for  understanding  how  the  Theatre  should  be  performing  and  highlights  the  gap  between  current 
and  required  performance.  It  may  also  be  used  to  monitor  the  impact  of  planned  and  unplanned 
change. 

In  this  paper  we  outline  the  aim  and  scope  of  the  study  in  section  1,  define  AST,  the  subject  of 
evaluation,  in  section  2,  discuss  the  aim  and  scope  of  the  study  in  section  3,  outline  the 
requirements  of  Theatre  evaluation  in  section  4,  compare  evaluation  methodologies  in  section  5, 
describe  the  Enterprise  Evaluation  Methodology  in  section  6  and  its  application  in  section  7,  and, 
finally,  present  a  brief  discussion  of  the  feasibility  of  the  evaluation  methodology  in  section  8. 

2  Definition  of  AST:  the  Subject  of  Evaluation 

AST  is  a  complex  modem  military  organization.  AST  was  raised  in  1997  to  be  the  principal 
Joint  Operational  HQ,  charged  with  the  planning  and  the  conduct  of  campaigns  for  the  Defence 


1  Senge,  Peter,  M.  The  Fifth  Discipline,  The  Art  and  Practice  of  the  Learning  Organisation.  (1992)  Random  House 

Australia 

2  Made  up  of  staff  from  the  Command  and  Control  Division  and  the  Defence  Systems  Analysis  Division 

3  From  the  HQAST  J8  Development  Branch 


of  Australia  and  its  interests,  together  with  the  development  and  refinement  of  campaign  doctrine 
and  processes  4. 

AST  is  described  as  all  the  resources,  capability  and  responsibilities5  at  the  operational  level 
delegated  to  COMAST  by  CDF.  In  2003,  prior  to  collocation,  this  encompasses  three  domains: 

1.  FIQAST,  which  has  overall  responsibility  for  planning  and  conducting  Joint  operations 
and  campaigns. 

2.  The  Component  Commands  (Air,  Land,  Maritime  and  Special  Forces),  Northern 
Command  (NORCOM)  and  joint  support  agencies  such  as  the  ADF  Warfare  Centre 
(ADFWC),  Australian  Joint  Intelligence  Centre  (ASTJIC)  and  the  First  Joint  Movement 
Group  (1JMOVGP). 

3.  The  Theatre  of  operations  where  the  Theatre  brings  into  effect  its  plans  by  creating  Joint 
Task  Forces  (JTF),  made  up  of  force  elements  sufficiently  prepared  to  perform  their 
assigned  tasks.  The  single  services  are  responsible  for  raising,  training  and  sustaining 
those  elements  so  they  can  deliver  the  required  level  of  preparedness.  The  joint  assigned 
force  outcome  is  the  effect  produced  in  the  Theatre  of  operations  as  a  result  of  the  Theatre 
command  processes  and  the  actions  of  the  force  elements  in  the  JTF. 

3  Aim  and  Scope 

The  aim  of  the  study  was  to  test  the  feasibility  of  providing  a  tested  evaluation  methodology  that 
would  enable  Theatre  to  develop  a  baseline  of  its  performance6.  The  Theatre  baseline  would  be 
a  suite  of  measurements  of  the  Theatre.  The  baseline  would  provide  an  initial  measurement  of 
the  current  state  of  the  Theatre  against  which  changes  over  time  could  be  compared.  The  future 
collocation  of  Theatre  FIQ’s  in  2007  was  highlighted  as  an  event  in  the  life  of  the  Theatre  that 
would  benefit  from  having  at  least  one  pre-move  and  one  post-move  baseline  conducted.  In 
addition,  once  Theatre  capability  was  known  using  standardized  measures  of  performance,  then 
the  benefits  of  new  war  fighting  concepts  and  future  capability  development  could  be  determined 
by  experimentation  and  reported  using  the  same  measures.  The  evaluation  methodology 
employed  for  the  baseline  was  also  expected  to  provide  a  mechanism  for  monitoring  the  ability 
of  the  Theatre  to  function  effectively. 

4  Requirements  for  Theatre  Evaluation 

A  number  of  criteria  were  identified7  that  defined  the  requirements  for  Theatre  evaluation. 
These  criteria  were  as  follows: 

•  Provide  information  on  how  Theatre  operates  with  respect  to  how  it  is  expected  to 
perform 

•  Provide  insights  into  how  Theatre  might  operate  differently  to  more  effectively  achieve 
its  goals. 

•  Be  relevant  to  all  activities  and  modes  of  Theatre  operations. 


4  Theatre  overview,  http://defweb.cbr.defence.gov.au/hqast/Overview.htm.  last  accessed  7  Mar  03. 

5  CDF  Directive  2/97  of  30  Jan  97,  describes  the  geographic  responsibility  of  the  Theatre  as  including:  Australia,  its 
territories  and  territorial  seas;  the  Australian  Fishing  Zone  (AFZ)  and  Exclusive  Economic  Zone  (EEZ);  the 
Australian  Search  and  Rescue  Region  (SRR);  regional  countries  involved  in  the  Defence  Cooperation  Program 
(DCP)  and  their  territorial  seas;  the  standing  operational  areas;  Australia’s  naval  control  and  protection  of 
shipping  (NCAPS)  area  of  responsibility;  intervening  international  waters;  and  the  airspace  above  all  of  these. 

6  This  paper  reports  the  outcomes  of  the  initial  phase  of  the  activity 

7  Richer  W,  Flayter  D.  Theatre  Evaluation  Development  Study  (2000)  FIQAST  Internal  Report. 


•  Be  relevant  for  different  Theatre  Tempo  (at  “rest”  or  involved  in  numerous  activities  and 
missions  simultaneously). 

•  Be  able  to  assess  Theatre’s  capacity  to  plan,  coordinate  and  conduct  concurrent  activities 
and  campaigns 

•  Consider  activity  within  the  Theatre  Headquarters  and  within  the  Theatre  of  Operations 
(assigned  joint  forces). 

•  Be  compatible  with  both  Single  Service  and  Strategic  level  evaluation. 

•  Support  development  of  the  Theatre  including  immediate  support  to  co-location  of 
HQAST  and  long-term  experimentation  programs. 

•  Contribute  toward  the  development  of  the  Theatre’s  operational  art. 

•  Have  long-term  relevance  and  be  repeatable  (longevity). 

•  Be  easily  understood  and  implemented  by,  ADF  Theatre  staff  and  incorporate  data 
collection  techniques  that  are  simple  to  perform  regularly  and  to  a  consistent  standard. 

•  Not  be  Theatre  resource  intensive. 

5  Comparison  of  Evaluation  Methodologies 

Faced  with  developing  a  suitable  evaluation  method  for  a  system  as  complex  and  dynamic  as 
AST,  three  types  of  models  were  examined: 

1.  Assigned  force  effectiveness; 

2.  Functional  decomposition  model; 

3.  Principal  component  model. 

5. 1  Assigned  Force  Effectiveness 

The  methodology  of  assigned  force  effectiveness  based  on  operations  research  techniques,  takes 
a  “black  box”  approach  to  measuring  system  performance.  The  internal  processes  of  sub 
systems  are  not  evaluated  directly.  Performance  is  measured  by  comparing  the  output  achieved 
with  the  inputs  applied  to  the  system.  The  advantages  of  the  methodology  are  that  it  is  simple 
and  direct.  However,  it  has  the  following  limitations: 

•  For  complex  and  evolving  systems,  measurements  made  at  different  times  may  be 
difficult  to  meaningfully  compare,  that  is,  it  is  not  longitudinal. 

•  The  environment  of  the  system  influences  measurements,  and  this  will  reduce  confidence 
in  generalisations  based  only  on  linking  outputs  to  applied  inputs. 

•  The  model  does  not  provide  any  direct  insights  into  mechanisms  operating  within  the 
boundaries  of  the  system. 

•  The  model  can  only  evaluate  the  system  in  operation,  not  at  rest. 

The  assigned  force  effectiveness  approach  was  judged  useful  for  evaluating  the  operations  of 
assigned  joint  forces,  but  less  useful  for  the  complex  and  evolving  Headquarters  component  of 
Theatre. 


5.2  Functional  Breakdown  Model 

The  Functional  Breakdown  model  decomposes  the  system  into  functional  elements.  It  assumes 
processes  or  tasks  can  represent  the  functions  of  the  system.  This  model  provides  more  insight 


than  the  “black  box”  model  into  the  internal  operation  of  the  system,  by  providing  direct 
measurement  of  functional  processes  that  collectively  represent  the  system. 


The  use  of  the  Australian  Joint  Essential  Tasks  (AS JETS)  for  evaluation  is  based  on  this 
approach.  The  functional  breakdown  model  was  used  in  conjunction  with  the  ‘black  box’ 
method  during  the  exploratory  observation  of  OP  GUARDIAN  II,  the  ADF  contribution  to  the 
Commonwealth  Heads  of  Government  Meeting  (CHOGM)  in  2002.  The  functional  breakdown 
model  provided  the  opportunity  for  analysts  to  identify  a  small  set  of  essential  tasks  based  on  the 
Op  GUARDIAN  II  objectives  The  ‘black  box’  approach  assisted  in  developing  within  each 
essential  task  Measures  of  Merit  (MoMs),  based  on  Force  Level  Effects,  for  Theatre  level 

o 

processes.  A  separate  paper  gives  full  details  of  this  work. 

The  NATO  Code  of  Best  Practice  is  based  on  a  functional  breakdown  model  applied  to  military 
operations8 9,  which  provides  guidance  to  approaches  in  measurement  of  C2,  Imagery  Surveillance 
and  Reporting  (ISR)  platforms,  missions,  considering  human  factors  and  capability  development. 
The  Code  presents  an  MoMs  hierarchical  model  widely  used  throughout  the  military  community 
in  the  assessment  of  military  and  system  effectiveness  and  performance.  Measures  of  force 
effectiveness  (MoFE)  are  derived  top-down  directly  from  mission  goals  and  requirements. 
Measures  of  effectiveness  (MoE)  are  a  decomposition  of  MoFEs  and  assess  the  operational  level 
military  goals,  with  Measures  of  Performance  (MoPs)  derived  from  MOEs  to  support  assessment 
of  internal  system  level  effectiveness.  The  results  of  evaluation  are  then  rolled  up  the  hierarchy 
to  inform  on  the  overall  MoFE.  Process  decomposition  into  Task  Lists,  such  as  the  United  States 
(US)  Universal  Joint  Task  List  UJTL10,  is  the  foundation  of  most  task  based  military  assessment 
systems  with  allies. 

The  advantages  of  the  functional  break  down  model  are  that  it  provides  a  direct  measure  of  the 
system  components  and  that  it  has  more  flexibility  than  the  force  effectiveness  model  in 
accounting  for  differences  over  time.  The  limitations  are  that  it  assumes  that  the  overall  system 
performance  is  an  aggregate  of  functional  performance.  Improvements  in  a  functional  unit’s 
performance  will  improve  overall  system  performance.  The  model  does  not  directly  account  for 
the  interaction  at  functional  interfaces,  an  important  consideration  in  a  complex  organization 
such  as  AST.  Moreover,  while  the  model  focuses  on  the  tasks  or  processes  of  the  organization,  it 
does  not  take  into  account  the  values  or  principles  that  drive  the  organization. 


5.3  Principal  Component  Model 

The  Principal  Component  model  does  take  into  account  the  values  and  principles  that  drive  an 
organization.  It  is  an  approach  to  evaluation  based  on  the  values  underpinning  the  functioning  of 
the  enterprise.  This  methodology  relies  on  accumulating  a  comprehensive  collection  of 
elemental  descriptive  statements  covering  all  aspects  of  the  system.  The  principal  components  of 
the  system  can  then  be  determined  by  aggregating  the  elements  into  groups  that  are  as  far  as 
possible  orthogonal  (i.e.  minimal  cross  influence  between  groups)  to  each  other.  It  is  these 
components  that  can  then  be  evaluated. 


8  Chisholm  J  &  Richer  W.  Exploratory  Observation  of  Operation  GUARDIAN  II  (CHOGM  02),  DSTO  Technical 
Note  in  preparation. 

9  NATO  Code  of  Best  Practice  (COBP)  for  C2  Assessment  (RTO-TR-073  Revised  2002)  and  NATO  COBP 
Analysis  of  Small  Scale  Contingencies  (SAS-027). 

10  US  UJTLS  3500.04B  1999  Version  4. 


The  advantages  of  the  Principal  Component  model  are  that  the  process  itself  generates  a  deep 
level  of  understanding  of  the  system.  The  selection  of  appropriate  strategic  goals  leads  to  an 
evaluation  method  that  goes  beyond  the  process  bias  of  the  other  two  evaluation  methodologies. 
The  principal  components  that  emerge  from  the  process  identify  the  principal  factors  that 
contribute  to  the  performance  of  the  system.  Finally,  there  is  a  documented  link  back  from  the 
performance  measures  of  the  principal  components  to  each  of  the  large  number  of  specific  sub 
components. 

The  limitations  are  that  the  model  relies  heavily  on  the  selection  and  understanding  of 
appropriate  goals  of  the  system,  and  the  selection  of  an  appropriate  method  for  generating  a 
comprehensive  set  of  elemental  statements  describing  all  aspects  of  the  system.  It  requires 
substantial,  and  committed  subject  matter  advisor  involvement.  It  also  requires  intensive  initial 
resource  input  to  create  the  element  database  and  to  detennine  the  principal  components. 

The  Principal  Component  model  was  adopted  for  this  study,  evolving  into  what  we  have  called 
the  Enterprise  Evaluation  Methodology  (EEM).  In  the  next  section  we  describe  the  EEM. 

6  The  Enterprise  Evaluation  Methodology 

The  EEM  is  an  evaluation  model  designed  to  facilitate  the  extraction  of  the  Theatre’s  current 
strategy  into  performance  measures.  The  model  seeks  to  identify  activities  that  must  be 
performed  to  a  certain  standard  for  the  Theatre’s  strategy  to  succeed.  In  producing  an  evaluation 
of  Theatre  from  a  strategic  focus  the  EEM  ensures  that  developments  of  elements  within  the 
Theatre  will  not  compete  for  resources,  but  allow  allocation  in  a  coordinated  way.  The  EEM 
demonstrates  how  the  many  measures  work  together  and  their  cause  and  effect  relationship  with 
the  implementation  of  Theatre  strategy. 

Figure  1  illustrates  the  sequence  of  events  involved  in  the  EEM.  Four  phases  will  be  discussed: 

•  The  first  phase  translates  an  enterprise’s  strategy  into  an  enterprise’s  goals  fundamental 
to  achieving  that  strategy. 

•  The  second  phase  acknowledges  the  need  to  consider  the  many  interrelated  variables 
involved  in  the  functioning  of  a  complex  enterprise  system  and  employs  an  analytical 
enterprise  model  to  translate  the  enterprise’s  goals  into  performance  goals. 

•  The  third  phase  identifies  common  evaluation  measures  that  will  provide  data  on  more 
than  one  of  the  performance  goals  (to  reduce  the  resources  required  to  conduct  the 
evaluation) 

•  The  fourth  phase  of  the  EEM  takes  these  evaluation  elements  for  each  performance  goal 
and  identifies  metrics  and  means  of  measurement. 
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Figure  1: Enterprise  Evaluation  Methodology 


Evaluation  elements  provide  a  long-term  focus  for  Theatre  Evaluation.  For  a  new  enterprise 
strategy,  rather  than  the  evaluation  elements  changing,  it  is  the  link  between  the  evaluation 
element  and  the  performance  goals  that  would  be  expected  to  change.  In  the  EEM  model  each 
evaluation  element  represents  an  activity  of  the  enterprise,  which  is  common  for  more  than  one 
performance  goal. 

7  Application  of  the  Enterprise  Evaluation  Methodology 

In  the  previous  section,  the  meta-model  for  the  methodology  was  described.  In  this  section  we 
describe  the  application  of  the  methodology  utilizing  extant  strategic  war  fighting  guidance  for 
AST  and  an  enterprise  model. 


7.7  Phase  1:  Strategy  to  Enterprise  Goals 

The  first  phase  of  the  EEM  requires  the  involvement  of  the  enterprise’s  executive  to  select  the 
appropriate  strategy  and  then,  in  a  facilitated  workshop,  unpack  the  strategic  goals.  The  EEM 
seeks  to  identify  a  strategic  goal  for  the  business  of  war  fighting.  For  this  study  the  war  fighting 


strategy  of  Decisive  Manoeuvre  (DM)  as  described  in  document11  was  selected.  DM,  consists  of 
five  core  warfighting  concepts  12  and  four  supporting  concepts  13  that  underpin  and  enable  the 
overarching  concept  of  DM.  These  “Enablers”  embody  clear  goals  and  provide  a  vision  for  the 
Theatre  at  the  operational  level,  as  it  works  towards  the  development  of  an  Australian  way  of 
war  fighting. 


7.2  Phase  2:  Enterprise  Goals  to  Performance  Goals 

The  EEM  requires  the  use  of  an  analytical  enterprise  model  that  best  represents  the  enterprise 
and  focuses  on  its  strategic  development  initiatives.  The  enterprise  model  provides  a  framework 
within  which  a  team  of  senior  staff  are  able  to  review  the  defined  enterprise  goals  and  specify  the 
attributes  of  the  performance  goals  against  the  model. 

The  enterprise  model  chosen  was  the  MIT9014  model  because  of  its  familiarity  to  the  ADF  and 
Theatre  staff  and  its  application  in  related  DSTO  projects.  The  model  provides  a  framework  by 
which  to  consider  each  of  the  enablers  in  the  areas  deemed  by  the  model  to  be  necessary  to 
ensure  enterprise  success.  The  MIT90  model  proposes  the  following  areas;  technology, 
individuals  and  roles,  structure,  management  processes,  and  strategy.  The  definition  of  the  five 
areas  in  the  MIT90  model  are  as  follows: 

•  Technology:  the  Information  Technology  that  can  be  applied  to  facilitate  business 
processes. 

•  Individuals  and  roles:  concerned  with  people  within  the  organization,  tasks  they 
undertake,  and  the  education  and  training  they  require  to  perfonn  their  functions. 

•  Structure:  the  way  that  the  organization  is  partitioned  and  the  way  the  partitions 
interrelate. 

•  Management  processes:  the  standardized  sequences  of  activities  that  organizations 
adopt  in  order  to  undertake  the  tasks  they  perfonn  regularly. 

•  Strategy:  general  modes  of  doing  business  in  pursuit  of  organizational  goals15. 

The  MIT90  model  does  directly  account  for  the  interaction  at  functional  interfaces,  unlike  the 
functional  breakdown  models  discussed  previously. 

Performance  goals  were  defined  in  each  of  the  areas  for  two  representative  DM  Enablers.  The 
Enablers  selected  for  refined  unpacking  were  a  core  and  supporting  concept,  namely,  “Coordinated, 
Precise  Engagement  “(CPE)  and  “Decision  Superiority”(DS).  These  enablers  were  refined  to  produce  the 
approximately  130  performance  goals.  The  use  of  the  enterprise  model  is  illustrated  in  Figure  2. 


11  Connolly,  MG  J.M,  Decisive  Manoeuvre  (Jan  1998) 

http://defweb.cbr.defence.gov.au/hqast/i8/Items.%20References.%20Articles%20of)/o20General%20Interest%20to 

%20Staff.html.  last  accessed  7  Mar  03. 

12  Coordinated  Precise  Engagement,  Timely  Force  Projection,  Tailored  Sustainment,  Robust  Security  and  Effective 
Interoperability. 

13  Decision  Superiority,  Innovation,  Moral  Authority  and  Force  Preparation. 

14  Scott  Morton,  M.  The  Corporation  of  the  1990's:  Information  Technology  and  Organizational  Transformation, 
(1991)  Oxford  University  Press,  New  York 

15  Cook,  Kasser  and  Burke,  Assessing  the  C4ISR  Architecture  Framework  for  the  Military  Enterprise. 


Figure  2:  De-composing  and  translating  Decisive  Manoeuvre. 


THEATRE  GOALS 


Core  Concepts/Enablers 

1 .  Effective  Interoperability 

2.  Tailored  Sustainment 

3.  Robust  Security 

4.  Timely  Force  Projection 

5.  Coordinated  Precise 
Engagement 


Support  Concepts/Enablers 

1 .  Innovation 

2.  Decision  Superiority 

3.  Force  Projection 

4.  Moral  Authority 


iExample  Translation  of  an  Enabler  to  Performance  Goals 


Enabler 

Force  Projection 


Key  Planning 

Timely 
Implementation  Of 
individual,  unit 
training  &  appropriate 
upgrade  packages 


Example  Performance  Goals 

Individual  Training 

•  Have  done  a  Joint  Operations  Planning 
Course  before  sitting  on  Joint  Planning 
Groups. 

•  Ensure  Course  is  relevant  to  planning 
processes  in  Theatre. 

Applying  MIT90  Structure 

•  Structure-  a  place  responsible  for 
joint  training. 

•  Strategy- regular  JOPC’s  Regular 
JOPC  review/feedback  from  students. 

•  Process- Dcfinedprerequisites  for 
training  for  positions. 

•  Skill/Roles-  Suitable  background 

•  Technology-  Training  Databases, 
Operational  Command  Support 
Systems 


7.3  Phase  3:  Performance  Goals  to  Evaluation  Elements 

The  performance  goals  determined  in  phase  2  represent  critical  enterprise  attributes  and  these 
would  ideally  be  the  focus  for  developing  metrics.  However,  ADF  resource  constraints  require 
an  aggregation  of  evaluation  elements.  While  these  constraints  are  addressed  in  this  phase  of  the 
EEM  they  must  also  be  considered  in  the  development  of  metrics  and  means  of  measurement. 

This  phase  requires  the  identification  of  the  minimum  number  of  evaluation  elements  relating  to 
the  maximum  number  of  performance  goals.  Similarities  and  patterns  between  the  attributes  of 
Theatre  performance,  identified  during  phase  2,  must  be  found  so  that  multiple  attributes  can  be 
addressed  using  one  evaluation  element. 

It  was  not  possible  to  develop  the  performance  goals  for  the  two  Enablers  as  the  latter  were  not 
sufficient  to  provide  a  comprehensive  cover  of  all  performance  goals.  Therefore,  identifying  a 
minimum  number  of  evaluation  elements  that  could  infer  a  maximum  number  of  performance 
goals  was  not  possible  at  this  stage. 


7.4  Phase  4:  Metrics 

The  final  development  phase  of  the  EEM  takes  the  evaluation  elements  (in  the  context  of  their 
corresponding  performance  goals)  and  seeks  metrics  that  will  provide  data  on  more  than  one 
evaluation  element.  The  result  will  be,  where  possible,  the  minimum  number  of  metrics  (or 


actual  measures  that  need  to  be  taken  and  data  collected)  that  inform  on  the  maximum  number  of 
evaluation  elements. 


Figure  3  .'Metrics  architecture  for  TEC. 


Developing  the  metrics  for  the  evaluation  elements  requires  input  again  from  the  enterprise  staff, 
this  time  at  the  working  level.  Staff  with  both  specialist  knowledge  of  available  means  of 
measurement  and  the  activities  to  be  evaluated,  would  help  ensure  the  most  appropriate  means 
are  proposed,  tailored  to  the  current  situation  and  staffing. 

Finally  a  tailored  evaluation  plan  would  be  prepared  which  sets  out  clearly  the  metrics  along 
with  means  of  measurement  and  data  collection,  storage  and  reporting  procedures.  Most 
importantly,  to  enhance  the  enterprise’s  development,  the  plan  would  identify  the  relationship 
between  the  relevant  performance  goals  and  the  current  enterprise’s  strategy. 

In  the  exemplar  application,  a  metrics  architecture  was  developed  based  on  the  NATO  Measures 
of  Merit  (MoMs)  hierarchy.  This  presents  an  overall  metrics  architecture  which  consists  of 
Measures  of  Force  Effectiveness  (MoFE)  at  mission  goal  level,  Measures  of  Effectiveness 
(MoE)  supporting  MoFEs  at  a  lower  level,  Measures  of  performance  (MoPs)  at  the  system  level, 
and  conditions  and  standards  as  shown  in  Figure  3.  However,  the  actual  metrics  were  not 
developed  at  this  stage  because  of  time  constraints. 

8  Discussion:  Feasibility  of  the  Enterprise  Evaluation  Methodology 


The  adopted  EEM  has  proved  to  be  a  feasible  way  to  decompose  AST  Principles  (as  described  in 
DM)  into  performance  goals,  for  those  sample  enablers  selected.  The  application  exercise 
looked  at  the  viability  of  applying  the  model  for  Theatre  evaluation  and  started  work  on 
developing  evaluation  elements  and  thus  a  plausible  evaluation  instrument.  While  not  all  phases 
of  the  EEM  were  attempted  in  this  initial  stage  of  the  project,  it  was  partly  confirmed  that  the 
EEM  model  would  provide  a  means  to  develop  an  evaluation  instrument  for  the  Theatre,  going 
from  strategic  objectives  to  perfonnance  measures. 

Although  the  particular  approach  that  had  to  be  taken  was  resource  intensive,  the  approach  was 
judged  to  be  ahead  of  current  practice,  in  being  values  or  principles  based  rather  than  process 
based.  This  study  produced  a  methodology  that,  while  not  tested,  could  be  used  to  provide  a 
baseline.  The  next  phase  of  the  work  will  focus  on  further  the  development  of  the  evaluation 
methodology  in  the  context  of  a  comprehensive  Theatre  evaluation  capability. 
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Aim  and  Scope  of  Study 


•  Feasibility  of  Evaluation  Methodology 

•  Suite  of  Measurements  of  Theatre 

•  Initial  Baseline  to  Measure  over  Time 


AST  Subject  of  Evaluation 

•  Australian  Theatre  (AST)  Raised  in  1997 

•  Principal  Joint  Operational  HQ 

•  Planning  and  Conduct  of  Operations  and 
Campaigns 

•  Development  and  Refinement  of 
Campaign  Doctrine  And  Processes 

•  HQAST  plus  Component  Commands 

•  (Transformed  to  JOC  2003) 
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equirements  of  Theatre 
Evaluation 

How  Theatre  Operates  with  Respect  to  how  it  is  Expected  to 
Perform 


•  Relevant  for  Different  Theatre  Tempos 

•  Assess  Theatre’s  Capacity  To  Plan,  Coordinate  and 
Conduct  Concurrent  Activities  And  Campaigns 


Compatible  with  Both  Single  Service  and  Strategic  Level 
Evaluation 
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omparison  of  Evaluation 
Methodologies 


1 .  Assigned  Force  Effectiveness 

2.  Functional  Decomposition  Model 

3.  Principal  Component  Model 


Problem  Formulation 
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Select  &  Collect  Data 
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*Flood  and  Jackson,  1991 


Assigned  Force  Effectiveness 


•  Advantages 

-  “Black  Box”  Approach 
To  Measuring  System 
Performance 

-  Simple  And  Direct 


•  Disadvantages 

-  Measurements  Made 
At  Different  Times 
Difficult  To  Compare 

-  Environment 
Influences 
Measurements 

-  No  Insight  Into 
Mechanisms 
Operating  Within 
Boundaries  Of  System 


Functional  Breakdown  Model 


•  Decomposes  System  Into  Functional 
Elements 

•  Australian  Joint  Essential  Tasks  ASJETS 


Advantages 

-  Direct  Measure  of 
System  Components 

-  Can  Measure 
Differences  over  Time 


Disadvantages 

-  Cannot  Measure 
Interaction  at 
Functional  Interfaces 

-  Not  Values  or 
Principles 
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Principal  Component  Model 

Identifies  Elemental  Descriptive  Statements 

Aggregates  Elements  Into  Groups  to  Give 
Principal  Components  of  System 


Advantages 

-  Evaluates  Values  and 
Principles  That  Drive 
an  Organization 

-  Deep  Understanding 
of  System 

Disadvantages 

-  Selection  And 
Understanding  of 
Appropriate  Goals 

-  Comprehensive  Set  of 
Elemental  Statements 
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nterprise  Evaluation 
Methodology  EEM 


•  Based  on  Principal  Component  Model 

•  Extracts  Theatre  Strategy  into 
Performance  Measures 


•  Applied  a  soft  approach  based  on  MIT90 


EEM 


Phase  1 


Phase  2 


Phase  3 


Phase  4 
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Enterprise 

Strategy 


ENTERPRISE  GOALS 


Enterprise 
Model 


PERFORMANCE  GOALS 


Evaluation 

ELEMENTS 


METRICS 


Enterprise  Goals  for  Theatre 


•  Describes  5  interacting  areas  of  change 

-  Technology 

•  The  Information  Technology  that  can  be  applied  to  facilitate  business  processes. 

-  Individuals  and  roles 

•  Concerned  with  people  within  the  organization,  tasks  they  undertake,  and  the  education 
and  training  they  require  to  perform  their  functions. 

-  Structure 

•  The  way  that  the  organization  is  partitioned  and  the  way  the  partitions  interrelate. 

-  Management  processes 

•  The  standardized  sequences  of  activities  that  organizations  adopt  in  order  to  undertake 
the  tasks  they  perform  regularly. 

-  Strategy 

•  General  modes  of  doing  business  in  pursuit  of  organizational  goals. 
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Performance  Goals  to  Eval  Elements 


THEATRE  GOALS 

Decisive 

Manoeuvre 

(Joint  Warfare 
Concepts) 


Core  Concepts/Enablers 

1 .  Effective  Interoperability 

2.  T  ailored  Sustainment 

3.  Robust  Security 

4.  Timely  Force  Projection 

5.  Coordinated  Precise 
Engagement 


Support  Concepts/Enablers 

1 .  Innovation 

2.  Decision  Superiority 

3.  Force  Projection 

4.  Moral  Authority 


lExample  Translation  of  an  Enabler  to  Performance  Goals 


Enabler 

Force  Projection 


Key  Planning 

Timely 
Implementation  Of 
individual,  unit 
training  &  appropriate 
upgrade  packages 


Example  Performance  Goals 

^Individual  Training 

Have  done  a  Joint  Operations  Planning 
Course  before  sitting  on  Joint  Planning 
Groups. 

Ensure  Course  is  relevant  to  planning 
processes  in  Theatre. 


Applying  MIT90  Structure 

•  Structure-  a  place  responsible  for 
joint  training. 

•  Strategy- regular  JOPC’s  Regular 
JOPC  review/feedback  from  students. 

•  Process- Defmedprerequisites  for 
training  for  positions. 

•  Skill/Roles-  Suitable  background 

•  Technology-  Training  Databases, 
Operational  Command  Support 
Systems 


Evaluation  Element  to  Metric 
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Summarising  the  Process 


Examining  Decisive 
Manoeuvre  through 
the  MIT90  Lens 


9  Workshops 


Theatre  Evaluation  Methodology  (TEM) 


Evaluation  Product 


Example  Theatre  Health  Monitoring  Product 
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What  we  learned 


•  EEM  will  generate  useful  metrics 

•  There  is  a  cost: 

-  lot  of  effort  by  practitioners/S MEs  to  develop 

-  (Our  pilot  was  a  longitudinal  slice  for  Aus  Doctrine,  in 
the  face  of  restricted  resources) 

•  There  is  a  benefit  of  undertaking  EEM: 

-  practitioners/SME’s  learned  lessons 
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easibility  of  Evaluation 
Methodology 


•  We  have  described  the  “how”  method 

-  As  opposed  to  the  usual  “what”  method 

•  Method  needs  researcher  +  practitioner 

-  It  needs  solid  participation  by  those  within  the  process 
being  evaluated 

•  Issues 

-  Generation  of  metrics  is  resource  intensive 

-  EEM  models  a  changing  system  -  can  it  catch  up? 

-  Evaluation  undertaken  in  this  way  is  an  intervention 

•  Ongoing  development  and  validation  are  needed 


